This disclosure relates to a system for registering and tracking the position of a person's eye, in particular for refractive ophthalmic surgery, comprising:                a camera system (18) for taking images of the person's eye;        storage means connected to the camera system (18) for storing at least one eye image as a reference eye image;        an image processing system connected to the storage means and the camera system (18) for comparing a momentary eye image with the reference eye image and for outputting a signal representing a change of eye position between the reference eye image and the momentary eye image.        
Such systems are e.g. used in refractive ophthalmic surgery, i.e. in surgical operations in which the cornea of a patient's eye is shaped by a laser beam in order to correct for defects of vision. Before the surgical operation, a measurement of the patient's eye is made with the patient usually sitting in an upright position while focussing on a target image. A so-called wavefront analyzer or other refractive diagnostic device, such as corneal topographer or refractometer, then objectively determines an appropriate wavefront correction for reshaping the cornea of the eye. Typically the wavefront analyzer calculates a cylindrical or quasi-cylindrical ablation profile, which is to be applied to the eye by means of a focussed laser beam.
During the refractive surgery operation, the patient usually lies on his back, i.e. his head is typically in another position than during diagnosis with the wavefront analyzer. In order to correctly apply the previously calculated ablation profile to the eye, it is therefore essential to correctly register the position of the person's eye, i.e. to determine the translational and rotational displacement of the eye with respect to its position during diagnosis, so that the system “knows” how to apply the previously calculated ablation profile to the cornea in its momentary position.
Not only changes of eye position between diagnosis and surgery, but also eye movements during the operation have to be taken into account. In refractive laser systems, the laser operates in fixed space coordinates. The patient's eye is classically stabilized by voluntary visual fixation of a visual target. However, eye movements cannot be eliminated completely by voluntary fixation and furthermore slower head movements still occur during the surgery procedure, both changing the eye's position relative to the laser. However, with the increasing demands of customized corneal ablation, in particular the use of smaller beam sizes, faster repetition rates and greater precision of correction, exact positioning of each single laser shot onto the eye has become increasingly more important. This need for greater positioning accuracy has provided the impetus for several refractive laser companies to implement eye tracking systems into their surgical systems in order to position the ablation beam accurately onto the corneal surface and to compensate for patient head and eye movements during the operation. Many eye tracking systems track the image of the pupil. However, the exact position of this image depends on the refraction of light through the cornea in front of the pupil. The amount of corneal tissue through which the light passes may change due to the orientation of the eye, leading to an artificial shift of the image position, which negatively affects the tracking.
When markers are applied to the eye and tracked instead of or in addition to the pupil, other problems arise: The markers may irritate the eye or require anesthesia. In other cases, the attachment of the markers to the patient's eye may not last much longer than for example one hour, therefore imposing a time limitation to the operation and permissible time between refractive measurement and correction
Document WO 01/78584 A2 discloses a system for registering and tracking the position of a person's eye according to the preamble of claim 1. For registering purposes, this system compares the limbus centers of the eye in the reference eye image corresponding to its position during diagnosis and in the momentary eye image, and calculates a translational displacement of the eye from the difference of the limbus centers. Furthermore, the position of other eye features like e.g. retinal blood vessels, scleral blood vessels or a retinal nerve are compared between the reference image and the momentary image in order to determine the rotation of the eye between its diagnosis position and its surgery position.
In this prior art system, the eye is basically illuminated by daylight, and the cameras used to take pictures of the eye are “ordinary” cameras for collecting color images. All eye features necessary for registering are therefore visible in each image. However, as all eye features are always illuminated by daylight, the contrast in particular of blood vessels is poor. The registering setup described in the above prior art document, however, is too slow for use in a clinical situation in which longer operation times result in less income, in particular due to the fact that the poor contrast of the essential eye features in all color images renders the localization of these features time-consuming.
Another limitation of the prior art system is that it does not differentiate between blood vessels that are stable relative to the eye (cornea) and vessels which are stable relative to the head. As it is possible for the patient's eye to rotate independently of the head, this leads to some vessels moving relative to the cornea (surgical target area) at varying rates. Blood vessels may move at different rates depending on the relationship between rotation of the head, rotation of the eye, depth of the vessel in the conjunctive or sclera and position of the vessel relative to the eye lids or cornea. Therefore the above described method may result in incorrect or inaccurate results, due to the non-differentiation between vessels that move with the eye, and vessels that move with the head.
It is therefore an advantage of the invention to provide a faster, more accurate system for registering and tracking the position of a person's eye.
According to some embodiments, this advantage is achieved by a system that allows images of the iris/pupil region of the eye to be taken at a different wavelength of light than images of the scleral blood vessels, so that both wavelengths can be set to suitable values in order to enhance the contrast of the respective eye feature.
As an example, in an embodiment of this system according to the invention the first wavelength lies in a wavelength range between preferably 810 nm and 880 nm, but can be also between 780 nm and 950 nm and the second wavelength lies in a wavelength range between preferably 520 nm and 570 nm but can also be 500 nm and 580 nm. In this case, the first wavelength corresponds to infrared (IR) light, which is known to be very suitable for enhancing the iris/pupil contrast. The second wavelength corresponds to green light enhancing the contrast of blood vessels.
In order to further increase the contrast of the scleral blood vessels, the second wavelength can be set to a local maximum of an absorption spectrum of hemoglobin. As is well known, this maximum lies at a wavelength of approximately 550 nm, corresponding to green light.
In order to produce the two kinds of images with different wavelengths, an embodiment of the system according to the invention comprises a first light source system illuminating the iris and the pupil of the eye and a second light source system illuminating the sclera of the eye. In this case, the first light source system emits light of the first wavelength, whereas the second light source system emits light of the second wavelength.
As to the question how these two light source systems co-operate and how the two kinds of images are made with light of different wavelengths, various approaches are conceivable:
In one embodiment of the system according to the invention, the first and second light source systems are controlled such as to simultaneously illuminate the respective parts of the eye. In this case, the patient's eye is simultaneously illuminated with light of two different wavelengths arriving from two different light source systems: The central part of the eye, namely pupil and iris are illuminated with a focussed IR beam, whereas the sclera of the eye is simultaneously illuminated with green light.
At least as far as eye tracking during a surgical operation is concerned, it can be assumed that the amplitude of eye movements is sufficiently small to make sure that no misalignment problems regarding this spatially structured arrangement of light sources will occur. Both light source systems can therefore be stationary. The spatial emission profile of the IR light source system then basically corresponds to a light cone having a diameter of approximately 10 to 11 mm, making sure that no IR light reaches the eye outside the limbus. In a similar way, the light emission profile of the second light source system must be designed such as to make sure that no green light of the second wavelength falls on the iris and pupil.
A particularly flexible adjustment of the spatially structured beams from the two light source systems can be obtained when the system according to the invention furthermore comprises light directing means for variably directing the light of the first light source system and/or the second light source system to the respective parts of the eye. As preferable examples of such light directing means, a scanning mirror and/or a movable lens can be mentioned. The use of such optical devices for precisely directing light beams on a target, in this case the region inside or outside the limbus, is well-known and will therefore not be described in detail.
As an alternative to the above-discussed example of a system using a spatially structured beam, another embodiment can comprise multiplexing means for controlling the first and second light source systems such as to alternately illuminate the entire eye. In other words, such an embodiment does not use spatially structured beams, but rather two separate larger beams of different “color” which are structured in time. As an example, the multiplexing means can choose the first light source system at first, so that the entire eye is illuminated with IR light. The camera system takes a momentary IR image of the entire eye, in which in particular the iris/pupil region has a strong contrast and in which almost no blood vessels are visible. This momentary IR image can be stored in the storage means. Then the multiplexing means switch from the first to the second light source system which then illuminates the entire eye with green light. Again a momentary image of the entire eye is made, in which in particular the blood vessels have a strong contrast. This image can also be stored in the storage means. The image processing system can then calculate any translational displacement of the eye with respect to the reference image based on the IR image of the eye allowing a precise localization of the pupil. Then a possible rotation of the eye with respect to the reference image can be calculated based on the green image allowing a precise localization of the scleral blood vessels.
In yet another embodiment of the invention, the image processing system is designed such as to subtract an image which is recorded at the second wavelength of light from the second light source system from a preceding image recorded at the first wavelength of light from the first light source system. The remaining “difference image” is even more dominated by the scleral blood vessels than the second image itself, which still enhances the precision of the blood vessel localization.
In view of a suitable pupil/iris contrast, the first light source system can be arranged such as to illuminate the eye at an angle of approximately 30° with respect to the visual axis. In particular an illumination of the iris and pupil from approximately 30° below the visual axis has turned out to be suitable in practical operation.
Whereas the first light source system, which basically has to produce one single IR light cone, can consist of a single IR light source, realization of the second light source system is more difficult, as it has to illuminate two separate regions of the eye, namely the sclera on the left and right side of the iris. In an embodiment of the invention suitable for fulfilling this illumination requirement regarding the second wavelength light, the second light source system comprises two light sources arranged such as to symmetrically illuminate the eye at angles of approximately +35° and −35°, respectively, with respect to the visual axis.
In order to further improve the quality of the images made, i.e. to enhance the contrast of the pupil/iris in the IR images and the contrast of the scleral blood vessels in the green images, additional measures can be taken in view of the camera system: Thus, in an embodiment of the invention, the camera system can be only sensitive to light of the first and the second wavelength. In this case, daylight in the operating room does not negatively affect the images.
In practice such an arrangement can be obtained when the camera system comprises one single camera provided with a double-passband filter. Alternatively the camera system can comprise a CCD camera having a combination chip with at least two separate color bands corresponding to the first and the second wavelength, respectively.
As yet another alternative, the camera system can comprise a first camera sensitive only to the first wavelength and a second camera sensitive only to the second wavelength.
Embodiments of the invention furthermore relate to a method of registering and tracking the position of a person's eye, in particular for refractive ophthalmic surgery, comprising the steps:                recording an image of the person's eye in a reference position;        storing the image as a reference eye image;        recording at least one momentary eye image;        calculating a change of eye position between the reference eye image and the at least one momentary eye image; and        outputting a signal representing the change of eye positioncharacterized in that the step of recording at least one momentary eye image comprises recording a first image containing the iris and the pupil of the eye at a first wavelength of light and recording a second image containing scleral blood vessels at a different second wavelength of light, and that the step of calculating the change of eye position comprises calculating a displacement of the scleral blood vessels between the reference eye image and the second image. As explained above, the first wavelength can be optimized in view of an optimum contrast of the iris/pupil for determining a translational displacement of the eye, whereas the second wavelength can be set such as to optimize the contrast of the scleral blood vessels for determining the rotational displacement. As explained above, the first wavelength therefore preferably corresponds to IR light, and the second wavelength preferably corresponds to a local absorption maximum of hemoglobin.        
In a further embodiment of the method according to embodiments of the invention, it comprises the step of extracting scleral blood vessels from the second image, and in still another embodiment it also comprises a step of classifying extracted blood vessels according to their stability and trackability. The subtraction step can be performed as has been described above, i.e. by subtracting the green image from the IR image. Other possible methods comprise selected enhancement/filtering, the use of matched filters and (multi)thresholding, the use of anisotropic filters and (multi)thresholding and in particular accumulated watershed segmentation. The watershed technique has a bias towards fully enclosed features. A method for removing this bias is to artificially add features to the image, such as a black grid, to enhance connectedness or circularity of the features, and then perform the watershed segmentation. The added artificial feature is then removed from the result, leading to a far less biased watershed segmented image. Repeated watershed segmentation with decreasing height level reveals increasingly finer vessels. This provides a stable estimate of the width of the vessels, since wider vessels create deeper valleys in the gray scale image and can be segmented in more height levels.
Criteria for the classification for the detected blood vessels comprise, among others, the question if a feature is a blood vessel and not for example an eye lash or other artifact, and if a blood vessel belongs to the sclera or to the conjunctiva. This classification can be done based on properties of the vessels, such as appearance, location, thickness, focus, connectedness of the vessels, vessel shape direction or contour and intensity/contrast or contrast changes along the length of the vessel. For example, it may be possible to distinguish blood vessels from an eye lash based on the straightness, length and direction (e.g. ±30° from the vertical) or focus of the feature.
As an alternative to the extraction and classification of blood vessels, the method according to the invention can comprise defining an area of interest in the reference eye image and calculating a maximum correlation coefficient between the area of interest and the second image.
According to a further embodiment the present invention deals with the problem of how to locate in an image of an eye those area or areas which contain blood vessels so that these regions can be used for either registration or eye tracking. While there are probably many areas in an eye picture which contain picture elements representing blood vessels, it would be helpful for the purpose of eye tracking or eye registration if one or more regions are chosen where the blood vessels are present in such a manner that they are particularly suitable for tracking or registration.
According to an embodiment of the invention there is provided a method for eye registration or eye tracking based on an initial or reference image and a momentary image, said method comprising: obtaining one or more of so called landmarks in said initial image containing image data which are likely to represent blood vessels or parts thereof; and based on said landmarks, selecting one or more regions of interest as parts of said initial image which are to be used for eye tracking or registration. According to this embodiment the landmark selection makes it possible to select areas (regions of interest) in the initial image which are particularly suitable for tracking or registration.
According to a further embodiment for each of said regions of interest, there is obtained a displacement measurement between said initial image and said momentary image; and if multiple regions of interest are used, said multiple displacement measurements are combined to obtain a final displacement measurement. This makes it possible to take into account that despite the selection of regions of interest which are particularly suitable for tracking, each individual measurement may be erroneous, and by using multiple measurements the accuracy can be increased.
According to a further embodiment the step of obtaining said landmarks comprises one or more of the following: performing a Fourier transformation based on said initial image and selecting as said landmarks pixels or groups of pixels which have a high intensity in the frequency domain;
convoluting said initial image with one or more templates representing an edge or a corner or a curve in order to select from said convoluted image or images such areas in which edge, corner or curve structures have been detected; or
calculating orthogonal gradients for said initial image and selecting the regions of interest based on said gradients calculated. The aforementioned methods of landmark selection make it possible to perform the landmark selection automatically. This makes it easier for the user to apply the method of the invention. The mentioned methods all give an indication where in the reference image there are contained structures which may be suitable for tracking, in other words they give an indication about where it is likely that blood vessels are present, and based on this indication there can then be selected the regions of interest in an automatic manner which is more convenient than a manual selection of the regions of interest. These methods are based on assumptions about how suitable blood vessels or parts thereof should look like, and the mentioned image processing methods are “sensitive” to such structures and therefore can be used for the automatic extraction of landmarks.
According to a further embodiment of the present invention there is provided a method which first calculates based on an initial or reference image two gradient images for orthogonal gradient directions. These gradient images give an indication of how strong or how steep the image intensity changes along the two orthogonal directions, and they therefore already give kind of a rough indication about the presence of blood vessels because blood vessels are areas where there should be some kind of image gradient in at least one direction.
The method according an embodiment of the invention then further performs a mathematical operation based on that two gradient images which makes sure that there is at least a minimum gradient in both of said orthogonal directions. This makes it sure that there is an image intensity change in both orthogonal directions, this means that there is some certainty that the image structure indicated by these gradients is not just for example a blood vessel extending only into the x- or the y-direction, but rather it is a structure which shows an intensity change (and thereby an image structure) along two orthogonal directions. This is particularly helpful because for the purpose of the detection of rotation angles for eye tracking or eye registration purposes the structure used for registration or tracking must be such that it is not only one-dimensional.
According to a particularly preferred embodiment the mathematical operation which ensures that there is a minimum gradient in both orthogonal directions uses a mathematical approach which ensures that the intensity change is independent from the coordinate system. For that purpose there is used a covariance matrix based on the two gradient images. There is formed a covariance matrix for a certain predefined window around each pixel of the reference image, and based on this covariance matrix the eigenvalues are calculated. These eigenvalues represent the image gradients in two orthogonal directions in a manner independent of the coordinate system, i.e. in a covariant manner.
By taking the minimum eigenvalue for each predetermined window around each pixel of the reference or initial image it can be made sure that the thus selected eigenvalue gives an indication about the minimum gradient of the image in both two orthogonal directions.
The aforementioned method can be applied for all pixels of the reference image, thereby obtaining a minimum eigenvalue for each of the pixels of the reference image, whereas each minimum eigenvalue thus calculated gives a representation about how strong at least the image gradient is in two orthogonal directions. Because especially for the purpose of rotation measurement it is important that the selected features (and the regions of interest selected which contain the suitable features) have gradients in both orthogonal directions to make a rotation measurement possible. This can be achieved by the aforementioned method of using the minimum eigenvalues calculated for each of the pixels of the reference image.
Based on the thus obtained eigenvalue image according to one embodiment one can then select those areas of the reference image which can be assumed that they contain particular suitable structures (blood vessels) for the purpose of tracking or eye registration. These regions (or even individual pixels) may be called landmarks, and they are selected based on the minimum eigenvalue image. For example, one can at first choose which has the maximum eigenvalue among the eigenvalue image pixels, and then define a region of interest around this selected pixel. This will then give the first region of interest for the purpose of eye tracking.
After having blanked out the thus selected pixels of the first region of interest one can again go through the eigenvalue image of the reference image and can select the next highest eigenvalue. Around this pixel one can form a second area of interest, and the procedure may then be repeated either until a suitable number of areas of interest has been obtained, or for example until the eigenvalue corresponding to a certain pixel of the reference image falls below a certain threshold.
Of course it is also possible to imagine other ways of selecting or extracting so-called “landmarks” in the initial image, where said landmarks can be assumed to contain suitable blood vessel structures. One can for example take groups of pixels, e.g. of 5×5 block size from the eigenvalue image, calculate their average intensity, and based on these values select those blocks for which the average intensity is relatively high, e.g. by applying a threshold, by selecting the n blocks with the highest average intensity, or the like.
The thus selected “landmarks then form the base for the “regions of interest” which are then used for a comparison between the reference image and the momentary image for tracking or registration. The regions of interest typically are areas, which are chosen such that they surround the selected “landmarks”, e.g. by forming a predefined window around them.
According to a further embodiment one can then use the thus obtained predetermined regions (areas of interest) for the purpose of displacement measurement. For that purpose one compares each area of interest in the reference image with a momentary image to look how much the area of interest must be shifted to find again the area of interest in the momentary image (to look for a “match”). This can for example be done by calculating a correlation measurement for each shifting or displacement value within a certain predetermined window around the region of interest, and this then will lead to a map or an “image” of correlation values where each pixel corresponds to a certain displacement of the reference image and indicates the correlation measurement for this displacement.
According to a preferred embodiment the calculated matching score (the correlation measurement) is further weighted by a weighting value which indicates the “directionality” of features in the reference image like blood vessels. This can for example be done by applying anisotropic steered filters such as a bank of Laplacien of Gausian (LoG) filters. This will then given an indication about the “directionality” of features for each of the pixels of the reference image, and it will kind of “enhance” those structures for which there is a strong directionality like in the case of blood vessels which are long and slim in shape, i.e. have a strong directionality.
Based on the weighting then there is obtained a matching score map for each of the regions of interest, whereas each pixel of the matching score map indicates the matching score for a particular displacement of the reference image based on the correlation value calculated for this shift and weighted with the weighting map.
According to a preferred embodiment the matching scores are then accumulated for the individual displacement values and for the multiple regions of interest to thereby obtain a shift value which is most likely to represent the actual shift value.
This accumulation of multiple matching score maps takes into account and to some extent corrects several effects which may negatively influence the measurement for the individual regions of interest. For example, individual blood vessels may shift their position independently of the eye movement just due to their instability. Moreover, the measurement itself may be erroneous. These effects can be at least to some extent be taken into account and be corrected by accumulating the matching scores for the individual regions of interest.
According to a preferred embodiment furthermore an a priori knowledge about the correlation coefficient and the probability that the eye displacement actually takes the measured value is used to replace the correlation measurement by a corresponding probability. Using this probability thus obtained there is then calculated the accumulated probability for each of the individual displacements based on the multiple regions of interest for which the correlation map has been calculated.
This will then finally give a maximum probability for one of the displacement values which can then be taken as the final displacement value obtained from the measurement.
According to a further preferred embodiment the imposition uncertainty which is introduced by the measurement error is also taken into account by computing for each position the accumulated probability of its neighbors.
According to a further embodiment it is also possible to further classify the selected landmarks or regions of interest according to this suitability for tracking. This can be done by any known classification method or method of supervised learning like neural networks. Such classification techniques or supervised learning techniques may also be themselves used for the selection of the landmarks or regions of interest. This can e.g. be done by successively classifying regions of interest in the initial image using a classifier such as a neural network or any other method of supervised learning and classification.